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Verification and validation are essential aspects of mathematics and beyond in STEM, but these 
constructs are not consistently defined in research nor in curricula documents. In this theoretical 
paper, we argue that verification and validation are largely characterized as binary judgments by 
teachers and researchers about what students do. We then present empirical examples of student 
work to show this view does not account for students’ thinking as they resolve problems. We 
conclude that in order to foster learners who are confident and capable in STEM fields, it is 
necessary to revisit how verifying and validating activities are conceptualised and developed 
across years of schooling. 


International calls continue for a greater focus on STEM education and an increased 
emphasis on mathematics to meet social and economic challenges into the future (English, 
2016). Advocates of STEM agendas have championed “well-developed curricula that 
concentrate on twenty-first century skills including inquiry processes, problem-solving, [and] 
critical thinking” as well as content knowledge (English, 2016, p. 3). Recent trends have called 
for studies of how to leverage student achievement in one area to support similar gains in others, 
especially where subjects are naturally integrated (e.g., use of simulation apps and computer 
coding in conjunction with mathematical modelling to gain understanding of a real-world 
problem such as scheduling of in-patient transport in hospitals). As mathematics needs to play 
more of a foundational role accessing key concepts and providing investigative tools for 
interdisciplinary problems (Marginson et al., 2013), many curriculum authorities have 
responded by advocating an increase in mathematical modelling (e.g., National Governers 
Association Center for Best Practice & Council of Chief State School Officers 
[NGACBP&CCSO], 2010). Indeed, Sokolowski (2015) has confirmed that mathematical 
modelling activities generate positive learning effects when compared to other teaching 
methods in any mathematical content domain. As modelling both relies on, and fosters, many 
critical thinking skills identified as 21‘ Century Skills desired for daily life (English & 
Gainsburg, 2016), this is not surprising. 

Yet, classroom modelling does not approximate professional mathematical modelling with 
regard to verifying a model or validating the modelling. This capability is fundamental to all 
STEM disciplines as mathematical models are developed based on aspects of the real world 
that modellers come to understand are valued by their clients. This involves modellers using 
their prior knowledge of the real world that impinges on the problem being modelled, 
researching the context of the situation they need to model, as well as mathematical knowledge 
when formulating a model and when verifying and validating the model(s) constructed or 
applied. In contrast, classroom modelling is usually developed based, at least partially, on 
pedagogical concerns. As recommendations for both curriculum and teachers shift towards 
eliciting and building on students’ ways of reasoning (NCTM, 2015), expectations on students 
and teachers are evolving especially with regards to how students are to verify and validate their 
models. 


2018. In Hunter, J., Perger, P., & Darragh, L. (Eds.). Making waves, opening spaces (Proceedings of the 
41* annual conference of the Mathematics Education Research Group of Australasia) pp. 250-257. 
Auckland: MERGA. 
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In this paper, we will examine existing research, curriculum documents, and student activity 
to argue that there is inconsistency in the meanings of verification and validation in both 
research and educational literature that prevents development of robust and rigorous criteria for 
studying these important constructs and developing them fully in classrooms. In particular we 
argue that curriculum documents reflect the inconsistency found in research literature and 
therefore may not support teachers in developing students’ verification and validation skills. 
Using examples of student work, we then illustrate various aspects of verification and validation 
that may be overlooked with simplistic or broad definitions. Our contention is that for the field 
to move forward, it is necessary to stop asking questions that can be answered dichotomously, 
such as: Did the student validate the model? Instead, we propose asking questions that enable 
documenting students’ thinking and metacognition such as: How did the student validate the 
model? 


Verification and Validation in Curriculum Documents 


In order to ascertain what messages about verification and validation are conveyed in extant 
curriculum documents, a selection of these were analysed. In Australia, there is a national 
curriculum, but states are responsible for education. In Victoria, the current curriculum is 
presented in the Victorian Curriculum Mathematics: F-10A (VCAA, n.d.) and the Victoria 
Certificate of Education: Mathematics (Grades 11-12) (VWCAA, 2015). Similarly, in the United 
States of America (USA), a guiding national document, The Common Core State Standards: 
Mathematics (NGACBP&CCSO, 2010), exists but local jurisdictions are responsible for the 
curriculum. Texas, for example, has its own competence-based standards, the Texas Essential 
Knowledge and Skills (TEKS) (SBOE, n.d.). TEKS address all grades although in Grades 9-12, 
students select to study mathematics with a particular content focus (e.g., Algebra I or Pre- 
calculus). 

Recognising that important aspects of mathematics should be included in curricular 
documents in order to be subsequently valued and fostered in the classroom, Kim and Kasmer 
(2006) analysed 35 USA state standards documents (Grades 0-9) to determine whether they 
supported key aspects of reasoning. Reasoning for verification was the researchers’ primary 
focus. A list of keywords related to reasoning for verification were used to analyse grade-level 
expectations expressed in the documents. The analysis found that verification was expected 
mainly in upper grades. Particular tools were expected to be used to verify results (e.g., 
calculators). Other instances of expected verification were to verify predictions, conclusions, 
solutions, and mathematical relationships and ideas. 

Drawing on the methodology of Kim and Kasmer (2006), we selected the terms, 
valid/validate and verify, and additionally considered terms that might be indicative of 
verification and validation. The key terms selected for our analysis were: compare, check, 
dimensional analysis, draw conclusions, estimate, ideal, justify/ justification, limitations, 
predict, reasonableness/ reasoning, reflect, special/extreme case and valid, validity, and verify. 
Where terms have multiple meanings, only those where the meanings could be seen to be 
supporting verification and validation were included (i.e., comparing attributes or geometric 
reflections was excluded). All authors coded at least one document and then these were cross- 
checked and collated by the third author. 

Table 1 shows at what grade levels the key terms were identified in the TEKS (SBOE, n.d.). 
Terms that did not appear were excluded. Across the TEKS, the focus is on validation and 
verification of mathematical results. Students are expected to determine the reasonableness of 
solutions and select appropriate models, but not to develop, construct, or adapt models. They 
are expected to justify, compare, and assess reasonableness. 


251 


Table 1 
Grades and Subjects where Key Terms Identified in Texas Curricula 


Key Term K 12 3 4 5 6 7 8 Al A2 GP MQIS D S° AR 
Compare v vvvV Vv v vv vv v 
Draw conclusions V V Vv Vv vv Vv v 

Estimate VvVNVVNV VT VV VC VV VVV VV 
Justify VvVvVANVVV VV VV VT VC VV VVV VV VW 
Limitations vv v 

Predict VvVANV—V VV VV VV VC VV VVV VV VW 
Reasonableness vVvvVvvV VV VV VV VT VC VV VVV VV VW 
Valid/validate v vv 

Verify v v v 


Note. Al Algebra I, A2 Algebra I, G Geometry, P Pre-calculus, M Mathematical models, Q Advanced quantitative 
reasoning, IS Independent study, D Discrete mathematics, S Statistics, AR Algebraic reasoning. 


In Victoria (see Table 2) students are expected to assess reasonableness of estimates, 
answers, and results. In upper secondary, they verify results and solutions and test the validity 
of conclusions, arguments, and models. There are many examples related to use of technology 
such as to “relate the results from a particular technology application to the nature of a particular 
mathematical task (investigative, problem solving or modelling) and verify these results” 
(VCAA, 2015, p. 36). What is missing is a viable description of how students are supposed to 
make these judgments and against what standards. 


Table 2 

Grades and Subjects where Key Terms Identified in Victorian Curricula 
Key Term Fl23 4 5 67 8 9 10 F G M S FM MM SM 
Compare VV Vv VV VV VV VV VW v 
Check v vv v v 
Draw conclusions v v v v v 
Estimate v vvVvvVv VV VV VT VV VV v v 
Justify v v v v v 
Limitations Vi Vv v 
Predict va v v 
Reasonableness v vv v VvVV Vv v v 
Valid / validate v vvv v v 
Verify v v ¥ vv v v v 


Note. F Foundation mathematics, G General mathematics, M Mathematics methods, S Specialist mathematics (Yr 
11), FM Further mathematics, MM Mathematical methods, SM Specialist mathematics (Yr 12), *10A. 


Verification and Validation in Research Literature 


In this section we briefly review a selection of research studies where verification and 
validation are a focus and then consider what could be the sources of conflicting findings. 

When a class of Victorian Year 6 students was introduced to mathematical modelling, there 
was little transfer of problem-solving techniques from other classroom mathematics 
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experiences to modelling, particularly checking (Brown & Stillman, 2017). The checking that 
occurred included questioning the validity of statements, judging reasonableness and logic of 
answers and using empirical testing to show correctness of a proposed solution. There were 
thus only the rudiments of verification and validation shown. In another primary school study, 
but in Japan, Kawakami (2017) reported that model validation triggered students’ combining 
of the models they constructed in their internal modelling world with the external model 
constructed in the real world through data generation and collection. In a secondary classroom 
context in Texas, assumptions made implicitly or explicitly in formulating a model based on 
student interpretations of problem contexts were reported as feeding forward into verification 
and validation activity (Czocher & Moss, 2017). Czocher (2016) found that students engaged 
in validating throughout their modelling activity, but that it was not solely focused on a 
prediction or result. Indeed, Czocher (2013) showed previously that there is important overlap 
between validation, verification, and metacognition. Stillman (2000) reported Year 11 students 
using prior knowledge of real world task contexts in verification activities by enhancing 
decision making and as a means of checking progress or judging the reasonableness of interim 
or final results. These studies show validation and verification occurs in classrooms but 
encompasses more than checking correctness of computation. In contrast, Blum (2015), 
drawing on several German studies, noted absence of validating in students’ solutions despite 
educational standards for mathematics requiring validating of mathematical results and 
checking, comparing and evaluating mathematical models with respect to the real situation. 
Over 80% of Year 6-11 students in a study by Ludwig and Reit (2013) did not validate the 
solution to better adjust their symbolic model to the given situation. 

What gives rise to such conflicting results? Firstly, a partial answer could come from 
previous work. Stillman, Brown and Galbraith (2010) indicate that student modellers in even 
lower secondary can overcome low intensity blockages to their progress whilst modelling by 
harnessing metacognitive activity including reflection on actions that allows rectifying errors. 
However, blockages of high intensity occur where the modellers resist accommodating new 
contradictory information resulting in cognitive dissonance (Festinger, 1957). Task solvers’ 
approaches to resolving cognitive conflicts in order to maintain their cognitive structure could 
be the key to whether or not verification and validation activities are manifest in the classroom 
and thus seen by researchers. Pseudo-learners, according to Raychaudhuri (2013), successively 
stockpile items of knowledge almost linearly making connections primarily from the context 
where the knowledge was taught. They do not recognize cognitive conflict as lack of connection 
means questions of conflict do not occur. However, they will compartmentalize the conflicting 
pieces if pointed out to them; so, the conflict will cause no perturbation to their cognitive 
structure and there will be no evidence of either verification or validation. 

Secondly, caution is needed in interpreting research results as there is no consensus in the 
use of terms such as verifying and validating with validating often being defined explicitly as 
only a last step and verifying of the model mathematically being implicitly understood to occur 
(e.g., Blum, 2015). Still others see verification as multi-faceted and likely to occur when any 
interim results are derived, or decisions taken that impinge on the models produced (Stillman, 
2000). Indeed, Czocher (2013) problematized the complex role played by validation in 
modelling as it accounts for ascertaining both the model-situation fit and verifying that its 
analysis was conducted correctly. Validation includes ensuring that the model is based on 
assumptions that represent real world problem constraints, and must therefore include interim 
checks, whether or not they lead to model revisions. 
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Empirical Examples 


In this section we present empirical evidence that whether verification and validation occur 
cannot be effectively treated as a dichotomous decision. We note that each example is drawn 
from research data. We argue that thinking of verification and validation as “checks” on interim 
or final results limits conceptualizing how these critical skills should be fostered. In each case, 
we problematize the questions: /s the model valid? and Has the student validated? 

Erin was calculating the daily cost of a food stall franchise at an international multi-venue 
event, as part of her modelling of a cost plan for the organisers. An internet search resulted in 
her choosing a total estimated cost per stall for the duration of the event. She calculated the 
daily cost using a 7-day week as $3600 per stall. Whilst recording further information about a 
venue, she suddenly recalled it operated for only four days. She inferred her daily cost estimate 
was incorrect and recalculated it as $6200. As she recorded the new result, she expressed her 
doubts of its correctness, so repeated the computation but obtained the same result. Having 
checked her intuitive doubt, she reluctantly accepted the result but sat for a moment, thinking 
it through again. This is an example of metacognitive awareness where a metacognitive 
experience (Flavell, 1979), an intuitive feeling that the result of the calculation was too large 
triggered the cognitive task of verifying the result. Without a benchmark to judge correctness 
in the real situation her feeling was intuitive. Intuitions can be a source of productive ideas, but 
they can hinder thinking and reasoning (Fischbein, 1987) so verification is necessary. On the 
surface this appears to exemplify merely checking by redoing a computation, but more is 
happening here for Erin. She verified her result, checking that the number made sense, a 
paragon of the objectives set by the curriculum. Thus, there must be more to verifying a 
prediction than checking it against real data or validating a model than checking whether it 
makes sense. 

Ari and Tony were working on how long it takes an average family to fill a wheelie bin that 
holds up to 48 kg of rubbish. Through an internet search they established an average Australian 
family produces 153.85 kg of waste weekly. Ari divided 153.85 by 48 on his calculator and 
stated: “It takes 3.2 days to fill a full bin.” Other group members challenged this, prompting 
him to repeat his computation and hold up his calculator to show them the result of 3.2. When 
Tony suggested 153.85 was 100%, Ari worked forward from 48 mentally estimating that 3.2 
by 48 gave 100% of 153.85. Tony was still unconvinced encouraging Ari to express his thinking 
to dispel this puzzlement. Tony argued that 7 days was to little time to make over 150 if 48 took 
3 days. Ari calculated 48 x 3.2, insisting: “To fill a full bin, it takes 3.2 days and gets to that 
[showing Tony his calculator] 153”. Tony tried to introduce a cognitive conflict for him by 
saying if it was 48kg for 3 days then it takes 3 times 48 for a week, which is 9 days not 7. Ari 
remained adamant: “but I just worked it out then”. Tony countered with: “There’s not 9 days in 
a week.” Others in the group supported Tony but Ari was convinced his calculation was correct 
as he had verified it by repeating it and doing the reverse by estimation and with a technological 
tool. He persisted with his way of thinking. As trying to facilitate cognitive conflict for Ari had 
not worked, Tony used a direct approach suggesting using 153.85 + 7 which he agreed was the 
waste per day and then 48 + 21.97 giving 2.2 days to fill the bin. Ari then conceded. 

According to the relevant curriculum, students in the context of “mental, written, and 
technology assisted forms of computation” are to “routinely use estimation to validate ... their 
answers” (VCAA, n.d., p. 65). This was of no help to Ari as he verified his calculations several 
times including using estimation. There is far more to be considered in real situations. Firstly, 
we can ask: Would researchers say Ari verified his model? Certainly, he checked the prediction 
was correct, but this did not lead to finding flaws or revising his model. Secondly, we can ask 
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about Tony’s role in the milieu. Tony experienced cognitive conflict when presented with Ari’s 
answer and was able to identify its source. He then tried, unsuccessfully, to provoke cognitive 
conflict for Ari. 

On another modelling task, Mance sought an expression for the quantity of buffering agent 
in a fish tank as a function of time, f. To do so, he needed to create a differential equation that 
would model the rate of change of quantity of buffering agent in the tank as a buffering solution 
entered and well-mixed solution left. The problem statement gave the concentration, C, of the 


buffering solution as 1 — e /60 g/L and the solution entering at a rate of 5 L/min. He assumed 
rates of liquid entering and leaving were equal and obtained “ =5(1- e/ 60) g/min. He then 


proceeded to validate that his model represented the real-world situation saying: “If you just 
multiply those two together, you’ ll have 5 times the buffering strength entering and that’d give 
you g/min. It’s asking for how much buffering is in it at any point in time. If you were to plug 
in a time for that you’d be multiplying for a minute rate. So I think, the strength of the buffering 
solution. Yeah, that’d be right. So, I think it’s C is equal to 5 times that. Because if you plug in 
time you’re gonna get an answer in grams and that’s what you want.” 

Mance’s equation was incorrect. It did not account for liquid leaving the tank nor for change 
in concentration of buffering agent in the tank. However, Mance validated his model checking 
it satisfied the question posed. A more appropriate way to view Mance’s work is to ask not 
whether he validated, but what aspect of the modelling process he validated, how he did so, and 
the sources of cognitive conflict that led him to validate. He used dimensional analysis to check 
his set up of the model and then examined whether he thought the model would lead to an 
answer. His prior academic experiences taught him to doubt his models until he checked the 
units but as they gave him the correct unit, he inferred incorrectly that his modelling was correct. 


Discussion 


Through analysis of curriculum documents, research literature, and empirical examples we 
have identified inconsistencies in how verification and validation are treated. The first 
inconsistency is conceptual and pertains to what object is verified or validated. In curriculum 
documents and in some research literature, validation is conceptualized as a check carried out 
at problem end. Indeed, some conceptualize validation as possible only if and after the student 
obtains a result (Ludwig & Reit, 2013). In our empirical examples, only Tony and Ari verified 
a final result. Erin verified an interim result and Mance validated the representativeness of his 
differential equation prior to solving it. The complexity of mathematical modelling presents 
many opportunities for errors or different ways of thinking about a problem and many 
opportunities for verification and validation. 

The second inconsistency is methodological. It is the grain size of validation and 
verification. Grain size determines what is observable and what is to be observed. Concerns 
captured by the questions: Js the model valid? Has the student verified or validated? correspond 
to a coarse grain size. As methodological or pedagogical questions, they are intended to observe 
the student’s final product and ascertain whether or not the answer is correct. They are 
evaluative questions with dichotomous answers. Coarse grain size analysis is unclear whether 
the student verified or validated in our empirical examples because not all results are correct 
but in all cases the student engaged in verifying activity in attempts to ascertain whether the 
model was adequate. Further, Ari and Mance convinced themselves that their models were 
adequate. In Erin’s work we observe her verifying an interim result by redoing her 
computations. In contrast, concerns captured by the questions: What is being verified or 
validated and how? have a fine grain size. In Tony and Ari’s discussion, we observe debate 
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over which result is correct. Mance used dimensional analysis to monitor his ongoing work, not 
because he suspected something amiss, but because he was confirming that “all is well” (Goos, 
2002, p. 286). Thus, fine grain questions are both more descriptive and more revealing of 
student thinking allowing for teacher intervention during modelling rather than waiting until 
the end. 

The third inconsistency is theoretical. Viewing validation and verification as dichotomous 
judgments that have a normatively correct answer ignores psychological and experiential 
aspects of the student as a rational actor. That is, because the dichotomous view emphasizes the 
end product of modelling over the model construction process, the locus of control for 
determining whether a model is adequate is external to the student. The brief analyses of the 
empirical examples and the review of related literature above suggest that multiple theoretical 
constructs are construed as impetus, means, and consequence of validating and verification 
activity. At minimum, an adequate theory of verification and validation should include 
constructs such as cognitive conflict, intuition, reflection, metacognition, and account for 
students’ prior experiential and academic knowledge. 

Shifting the view of verification and validation from a dichotomous judgment about a 
process or action to a richer description of students’ ongoing activity means encouraging 
learners and teachers to attend to more than success in resolving the modelling problem. It 
encourages attending to activities and skills that support learning how to carry out validating 
activities. These attendant skills would then replace the narrow product-oriented definitions of 
verification and validation currently found in curriculum documents. We do not suggest 
attending to the processes of modelling and verifying and validating activities in place of 
students obtaining correct answers. We advocate emphasizing these activities to foster learners 
who respond to cognitive conflict with restructuring of knowledge and changing their 
approaches rather than learners who resolve cognitive conflict via compartmentalization 
(Raychaudhuri, 2013). Finally, a descriptive rather than dichotomous view of verifying and 
validating activity begs the question: How can verification and validating activities be 
provoked? In the product-oriented dichotomous view the only sites for verification and 
validation occur as a result is reached and at problem end. Teacher moves might include asking, 
“Does the (final) answer make sense?” or pointing out an error in reasoning or judgment. As in 
the Ari and Tony example, just pointing to an error is often not sufficient to incur cognitive 
conflict necessary to provoke validating. A descriptive, fine grained view of verification and 
validating activities allows teachers to respond to students’ modelling activities not as though 
there is a single, high-stakes act of metacognitive decision making at the end but rather 
throughout the process. 


Conclusion 


We have shown that verification and validation are in fact more complex and nuanced 
activities than reflected by curriculum documents. Specifically, we have shown it is possible 
for students to engage in validating activity without arriving at a correct model or answer 
because it is possible to have any combination of correct or incorrect assumptions, models or 
results. Thus, if the only proficiency required is that students check their obtained results against 
some correct answer at the end of their resolution of the problem, teachers and researchers miss 
students’ natural ways of reasoning and how to build on them. As a research community we 
can shift away from asking dichotomous, evaluative questions towards richer questions that 
ground discussions of pedagogy in student reasoning such as: What is the student validating 
and how? and How can validation be provoked in this moment? In line with these conclusions, 
a richer view of verification and validation is necessary in order to align teaching and 
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assessment with the cognitive and metacognitive activities that support skills needed by 
successful STEM students. 
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